Features compared

Jürgen Schneider

25 September 2023

Why even bother?

specialized software organizes screening task
idea: ready-to-use setup, tailored at screening for reviews
save time
sort documents by relevance via machine learning

Software options

(Burgard & Bittermann, 2023)

Active learning

Example: SVM (support vector machines) used by Rayyan:

manually reviewed references word combinations that differentiate between included and excluded
predictive model computes inclusion probabilities to the remaining records

ASReview offers several algorithms to choose from

Output:
ASReview sorts the remaining records
Rayyan symbolizes inclusion probability as stars (and sorts)

How good does it work?

Stopping rules:

pragmatic: after …% of screened documents
heuristic: after … consecutive irrelevant documents
sampling: tool estimates to have found …% of relevant documents

Metrics:

Recall: Share of relevant papers found
89 of 100 relevant papers were found: recall of .89
WSS: Work saved over sampling (for a certain recall)
After screening 70% of documents, I found 100% of relevant records, therefore saved 30% of work

(Scherhag & Burgard, 2023)

How good does it work?

(Scherhag & Burgard, 2023)

How good does it work?

(Scherhag & Burgard, 2023)

How good does it work?

ASReview outperforms Rayyan
Stopping after …% most reliable
Combination of stopping rules
stopping after min. 80% screened AND 50 consecutive irrelevant

Practical features

feature	ASReview	Rayyan
Free	Yes	(Yes) Free version available
Open Source	Yes	No
Platform	User-unfriendly local installation	Browser-based or mobile version
Data remains on local machine	Yes	No

Practical features

feature	ASReview	Rayyan
Detecting duplicates	No	Yes
Two or more coders possible	No	Yes
# of docs to be coded before sorting algorithm can run	2 (1 exclude, 1 include)	50
Blinding authors possible	Yes	No
Keywords for include/exclude to be highlighted in abstracts	No	Yes (each coder)
Specific feature to code exclusion reason	No (needs to be typed in manually in the notes field for every doc)	Yes (once typed in, can be coded with other docs)
Feature to code labels	No	Yes (once typed in, can be coded with other docs)
Export of coded results in an easy to use/read form	Yes	No (but I can share a script for that)

Hands-on

Script for Rayyan

You’ll only need to change things in the lines with comments mentioning “DO:”

library(tidyverse)
library(rio)

freq_screened <- rio::import(file=here("[path to]/articles.csv")) # DO: INSERT PATH

freq_screened <- freq_screened %>%
  dplyr::mutate(excl_reason = str_extract(.$notes, "RAYYAN-EXCLUSION-REASONS:.*?(\\||$)"), # extract exclusion reasons
                labels = str_extract(.$notes, "RAYYAN-LABELS:.*?(\\||$)"),       # extract labels coded
                decision_c1 = str_extract(.$notes, "Jürgen\"\"=>\"\".*?\"\""),    # extract decision from coder 1, DO: CHANGE NAME!
                decision_c2 = str_extract(.$notes, "Juergen\"\"=>\"\".*?\"\"")) %>% # extract decision from coder 2, DO: CHANGE NAME!
  # remove prefixes or recode
  dplyr::mutate(excl_reason = str_remove_all(excl_reason, "RAYYAN-EXCLUSION-REASONS:\\s"),
                labels = str_remove_all(labels, "RAYYAN-LABELS:\\s|\\s\\|"),
                decision_c1 = case_when(                                      
                  str_detect(decision_c1, "Excluded") ~ "Excluded",           
                  str_detect(decision_c1, "Included") ~ "Included",           
                  str_detect(decision_c1, "Maybe") ~ "Maybe"),                
                decision_c2 = case_when(                                      
                  str_detect(decision_c2, "Excluded") ~ "Excluded",           
                  str_detect(decision_c2, "Included") ~ "Included",           
                  str_detect(decision_c2, "Maybe") ~ "Maybe"),
                agreement = case_when(        # is there agreement on the decision?
                  decision_c1 == decision_c2 ~ 1,
                  TRUE ~ 0)
  )

References

Burgard, T., & Bittermann, A. (2023). Reducing Literature Screening Workload With Machine Learning: A Systematic Review of Tools and Their Performance. Zeitschrift Für Psychologie, 231(1), 3–15. https://doi.org/10.1027/2151-2604/a000509

Scherhag, J., & Burgard, T. (2023). Performance of Semi-Automated Screening Using Rayyan and ASReview: A Retrospective Analysis of Potential Work Reduction and Different Stopping Rules. https://doi.org/10.23668/PSYCHARCHIVES.12843